Skip to content

feat: add macOS support and harden cross-platform memory scanner#23

Open
JeanExtreme002 wants to merge 14 commits into
mainfrom
jeanextreme002/macos-support-and-hardening
Open

feat: add macOS support and harden cross-platform memory scanner#23
JeanExtreme002 wants to merge 14 commits into
mainfrom
jeanextreme002/macos-support-and-hardening

Conversation

@JeanExtreme002
Copy link
Copy Markdown
Owner

Summary

Major release adding native macOS support via the Mach VM APIs, fixing latent bugs in the Windows/Linux backends, and tightening cross-platform robustness. Full notes in CHANGELOG.md.

Highlights

  • macOS backendtask_for_pid, mach_vm_read_overwrite, mach_vm_write, mach_vm_region, mach_vm_protect. Self-process works without entitlements; cross-process needs com.apple.security.cs.debugger or SIP off + root.
  • Critical fixes — platform detection ("win" in "darwin"), unchecked syscall returns, scan_memory off-by-one, PROCESS_TERMINATE being a silent alias of PROCESS_SUSPEND_RESUME (both 0x0800), default permission lowered from PROCESS_ALL_ACCESS to PROCESS_VM_READ, Linux MEMORY_BASIC_INFORMATION widened to 64-bit, WOW64-aware MBI layout on Windows, transparent mach_vm_protect flip on macOS writes to read-only pages.
  • Performance — 6-8× speedup on numeric scans via struct.iter_unpack + inlined per-scan_type comparison loops; NOT_EXACT_VALUE overlap check dropped from O(n·m) to O(n·log m) via bisect_left; multi-GB regions chunked at 256 MB across all backends.
  • API ergonomicsbufflength optional for numeric types (int/float/bool inferred), snapshot_memory_regions() + memory_regions= kwarg for refine-scan workflows, case_sensitive= exposed on OpenProcess, PyMemoryEditorError base class for all library exceptions.
  • Tooling — CI matrix expanded to 3 OSes × 6 Pythons (3.8–3.13), flake8 as a gate, mypy informational, pytest-cov reporting, Conventional Commits enforced on PR titles, Dependabot configured to suppress routine version PRs (security alerts still fire), auto-delete of merged branches.

Breaking changes

  • Default permission is now PROCESS_VM_READ; writers must opt in to PROCESS_VM_WRITE | PROCESS_VM_OPERATION.
  • PROCESS_TERMINATE value corrected from 0x08000x0001 (anyone relying on the buggy alias will see different behavior — they were terminating processes when asking to suspend/resume).
  • Python 3.6 and 3.7 dropped; minimum is now 3.8.
  • linux.ptrace and util.search (KMP/BMH) packages removed (never used in the scan path).

Test plan

  • flake8 PyMemoryEditor tests — clean
  • macOS (Tk 8.5 host) — 48 passed + 1 skipped in 0.08s on fast tests; full integration suite 53 passed + 3 skipped in ~15 min
  • Linux (Docker python:3.12-slim) — 70 passed + 3 skipped in 6.57s
  • Microbenchmarks confirm 6-8× speedup on BIGGER_THAN/SMALLER_THAN/VALUE_BETWEEN over 50 MB
  • Read-only page write on macOS exercised via mmap + mprotect in tests/test_macos_protect.py
  • WOW64 dispatch covered by mocked IsWow64Process test
  • AmbiguousProcessNameError and case_sensitive covered by psutil-mocked tests
  • Final CI run on this PR
  • Manual smoke of pymemoryeditor Tk sample on each OS with Tk ≥ 8.6

Major release adding native macOS support, fixing latent bugs in the
Windows/Linux backends, and tightening cross-platform robustness. See
CHANGELOG.md for the full list.

Added
- macOS backend via Mach VM APIs (task_for_pid, mach_vm_read_overwrite,
  mach_vm_write, mach_vm_region, mach_vm_protect). Self-process works
  without entitlements; cross-process needs com.apple.security.cs.debugger
  or SIP off + root.
- snapshot_memory_regions() + memory_regions= kwarg on search_by_value*
  and search_by_addresses for refine-scan workflows.
- bufflength is now optional for numeric types (int->4, float->8, bool->1).
- iter_region_chunks reads multi-GB regions in 256MB chunks across all
  three backends (prevents OOM in browser/JVM-sized targets).
- PyMemoryEditorError base + AmbiguousProcessNameError.
- py.typed marker; mypy + pytest-cov in CI.

Fixed (critical)
- Platform detection: "win" in sys.platform matched darwin, breaking
  macOS imports.
- Read/Write/process_vm_*v now check argtypes/return value; failed reads
  no longer return zeroed buffers indistinguishable from real data.
- scan_memory off-by-one (skipped last value of each region).
- scan_memory_for_exact_value NOT_EXACT_VALUE no longer yields every
  non-matching byte; aligned to target_value_size.
- WindowsProcess default permission: PROCESS_VM_READ instead of
  PROCESS_ALL_ACCESS (least privilege).
- Permission gate now requires VM_READ explicit OR all PROCESS_ALL_ACCESS
  bits set (was passing on any single bit).
- ProcessOperationsEnum.PROCESS_TERMINATE was 0x0800 — same as
  PROCESS_SUSPEND_RESUME — making it a silent Enum alias. Corrected to
  0x0001 per MSDN.
- Linux MEMORY_BASIC_INFORMATION fields widened to 64-bit (regions > 4GB
  no longer truncate).
- Linux /proc/<pid>/maps inode parsed as decimal (was hex).
- Windows MEMORY_BASIC_INFORMATION layout picked per target via
  IsWow64Process (no more corruption when 64-bit Python attaches to
  32-bit target).
- macOS write_process_memory to a read-only page now transparently
  elevates protection via mach_vm_protect and restores it.
- Linux scan filters shared mappings (parity with Win32/macOS).
- read_process_memory(addr, str, n) now decodes with errors="replace"
  to match convert_from_byte_array.

Performance
- 6-8x speedup on numeric scans via struct.iter_unpack + inlined
  comparison loops per scan_type. NOT_EXACT_VALUE overlap check is now
  O(log m) via bisect_left.

Tooling
- CI: 3 OSes (ubuntu/windows/macos) x 6 Pythons (3.8-3.13), flake8 gate,
  mypy informational, pytest-cov.
- Conventional Commits enforced on PR title (lint-pr-title.yml).
- Dependabot config (open-pull-requests-limit: 0; security alerts still
  fire).
- Auto-delete head branch on PR close.
- Tk sample now requires Tk >= 8.6 and aborts with platform-specific
  install hints when missing or outdated.

Removed
- Unused PyMemoryEditor.linux.ptrace package.
- Unused PyMemoryEditor.util.search (KMP/BMH never used in scan path).
- Python 3.6 and 3.7 support (minimum is now 3.8).
The default permission on WindowsProcess is now PROCESS_VM_READ (a 2.0
breaking change), so existing write tests in test_editor.py started
failing on Windows CI. Explicitly request VM_WRITE | VM_OPERATION on
Win32; Linux and macOS ignore the kwarg as before.
- Add mypy override that ignores errors in win32/linux/macos backends.
  Each backend uses symbols (ctypes.windll, WINFUNCTYPE, Mach types) that
  only resolve on their target OS; mypy running on one host can't validate
  the others.
- Cast() generic-return-vs-concrete-bytes/str in convert_from_byte_array.
- Loosen get_c_type_of return type to Any (returns either _SimpleCData or
  ctypes.Array without a common base).
- Tighten _as_bytes to only treat real bytes as a no-op (bytearray now
  goes through the bytes() conversion).
- Move GetProcessIdByWindowTitle import into the function body so mypy
  on non-Windows hosts doesn't see it as undefined.
test_search_by_int and test_search_by_float iterate every address yielded
by search_by_value_between and read it back to verify the value. A page
mapped at scan time can be decommitted/protected before the subsequent
read — the syscall now surfaces this as OSError (it used to silently
return zeros). Wrap the read in try/except OSError, matching the pattern
already used in test_search_by_string.
VirtualQueryEx requires PROCESS_QUERY_INFORMATION (or PROCESS_QUERY_
LIMITED_INFORMATION) in addition to PROCESS_VM_READ. With only
PROCESS_VM_READ — the previous default — VirtualQueryEx returns 0, so
get_memory_regions / snapshot_memory_regions / search_by_value* /
search_by_addresses all came back empty. Exposed by Windows CI when
test_region_snapshot opened a process without an explicit permission.

The new default is PROCESS_VM_READ | PROCESS_QUERY_INFORMATION (exposed
as the DEFAULT_PERMISSION constant). README and CHANGELOG updated.
tests/test_editor.py also adds PROCESS_QUERY_INFORMATION to its
permission combo to make the read-back loop in search-by-value tests
robust across Windows versions (it happened to work in Python 3.11 by
implicit grant, but the explicit bit makes it deterministic).
GitHub's macOS-latest runner pool is much smaller than ubuntu/windows.
Running 6 Python versions × macos in parallel stalls the PR for 10+
minutes in queue without acquiring a runner. macOS only validates that
the Mach backend works — the Python version doesn't change that surface,
so we drop down to a single (stable) Python on macos-latest while
keeping the full 6-version matrix on ubuntu/windows.
The macos-latest (Apple Silicon arm64) runner pool is heavily congested
on free-tier accounts — jobs sit in queue for 15+ minutes without
acquiring a runner. The macos-13 (Intel x86_64) pool is much larger
and exercises identical code paths: Mach VM structs are fixed-size by
design (mach_port_t = uint32, mach_vm_address_t = uint64, etc.), so
x86_64 and arm64 hit the same struct layout and syscall surface.
GitHub-hosted macOS runner pools are often congested and a job can sit
in queue for 30+ minutes without acquiring a runner. Stop the PR from
stalling on this:

- continue-on-error: true for macOS jobs (ubuntu and windows still gate
  the merge; macOS is best-effort)
- 25-minute timeout caps the wait; real test runs finish in well under
  10 min when a runner is available
timeout-minutes counts execution time, not queue time, so a macOS job
stuck waiting for a runner can stall a PR indefinitely. The fix:

- Split macOS into its own build-macos job
- Gate it with: if: github.event_name != 'pull_request'
- Runs on push-to-main, weekly cron, and workflow_dispatch — never blocks
  a PR

PRs are gated by ubuntu × 5 Pythons + windows × 5 Pythons + lint, which
already provides cross-platform coverage. The Mach backend is validated
by the merged code via cron/dispatch and by local self-process tests in
dev.
GitHub-hosted macOS runners are heavily congested on free-tier accounts.
Even with continue-on-error and timeouts, the job blocked the workflow
UI for tens of minutes per run. The Mach backend is covered by local
self-process tests in dev; contributors with macOS hardware can run the
suite directly.
Drop PyMemoryEditor.sample (Tk) in favour of PyMemoryEditor.app, a
PySide6/Qt app that exercises every public surface of the library:
all eight ScanTypesEnum modes, the five value types
(bool/int/float/str/bytes), search_by_value / search_by_value_between
(with progress_information, writeable_only, region snapshot reuse),
search_by_addresses, read_process_memory, write_process_memory,
get_memory_regions / snapshot_memory_regions, plus value freezing and
a live hex viewer. Wired up as the pymemoryeditor CLI and a new
[app] optional-dependencies extra for PySide6.

Also extends the .flake8 ignore list with E203 (black-compatible
slice spacing, mirroring the existing W503 ignore), and updates the
README + CONTRIBUTING accordingly.
Black-style reformatting (slice spacing, line breaks around imports,
blank line after class docstrings, etc.) across the cross-platform
backends, util/, process/, the tests, and the two app modules the
editor revisited after the previous commit. No behaviour changes; the
.flake8 already tolerates E203 / W503 for compatibility with this
style, so make lint stays green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant